Leo Lu
(This material is modified from Mansun Kuo’s work)
2016-05-24
## === install required packages ===
pkg_list <- c("magrittr", "httr", "rvest", "stringr", "data.table",
"jsonlite", "RSQLite", "devtools")
pkg_new <- pkg_list[!(pkg_list %in% installed.packages()[,"Package"])]
if(length(pkg_new)) install.packages(pkg_new)
if("xmlview" %in% pkg_new) devtools::install_github("hrbrmstr/xmlview")
rm(pkg_new, pkg_list)All RStudio keyboard shortcuts
| Description | Windows & Linux | Mac |
|---|---|---|
| Attempt completion / Indent | Tab | Tab |
| Run current line/selection | Ctrl+Enter | ⌘+↩︎ |
| Comment/uncomment current line/selection | Ctrl+Shift+C | ⌘+⇧+C |
| Reindent lines | Ctrl+I | ⌘+I |
| Insert pipe operator | Ctrl+Shift+M | ⌘+⇧+M |
?: Access ducument in R console??: Search the halp systemCheck your working directory everytime you start to work!
Using getwd/setwd to get/set your working directory.
(wd = getwd()) # current config
setwd("resources/img")
getwd()
setwd(wd)RStudio will set working directory automatically when opening new files.
If you use Projects, RStudio will change working directory for you automatically.
Vector, Matrix, Array, List and Data frame are the most basic data structure in R. These data structures can be mapped into a table according to:
| Homogeneous | Heterogeneous | |
|---|---|---|
| 1d | Atomic vector | List |
| 2d | Matrix | Data frame |
| nd | Array |
v1 <- c(1:10)
v1
#> [1] 1 2 3 4 5 6 7 8 9 10
is.vector(v1)
#> [1] TRUE
length(v1)
#> [1] 10
s1 <- 2
s1
#> [1] 2
is.vector(s1)
#> [1] TRUE
length(s1)
#> [1] 1Lists are also vectors, but not atomic vectors. Lists are generic vectors, with (naturally) different semantics.
Elements in a list can be any kinds type and its length is arbitrary.
Function
strcan help you investigate the structure of a nested list.
li <- list(a = 1:10,
b = c("apple", "banana"))
str(li)#> List of 2
#> $ a: int [1:10] 1 2 3 4 5 6 7 8 9 10
#> $ b: chr [1:2] "apple" "banana"
li2 <- list(li = li,
c = matrix(1:4, nrow = 2))
str(li2)#> List of 2
#> $ li:List of 2
#> ..$ a: int [1:10] 1 2 3 4 5 6 7 8 9 10
#> ..$ b: chr [1:2] "apple" "banana"
#> $ c : int [1:2, 1:2] 1 2 3 4
x1 <- list(c(1, 2), c(3, 4))
x2 <- list(list(1, 2), list(3, 4))
x3 <- list(1, list(2, list(3)))a <- list(a = 1:3, b = "a string", c = pi, d = list(-1, -5))[ extracts a sub-list. The result will always be a list.str(a[1:2])
str(a[4])[[ extracts a single component from a list. It removes a level of hierarchy from the list.y <- list("a", 1L, 1.5, TRUE)
str(y[[1]])
str(y[[4]])$ is a shorthand for extracting named elements of a list. It works similarly to [[ except that you don’t need to use quotes.a$a
a[["b"]]Data frame is a 2-dimension data structure to deal with a table-like heterogeneous data.
df <- data.frame(gender = c("male", "female", "female", "male"),
age = c(33, 18, 24, 26))
## Add new column in a data frame
df$city <- c("Taipei", "Taipei", "Hsinchu", "Taichung")df#> gender age city
#> 1 male 33 Taipei
#> 2 female 18 Taipei
#> 3 female 24 Hsinchu
#> 4 male 26 Taichung
str(df)#> 'data.frame': 4 obs. of 3 variables:
#> $ gender: Factor w/ 2 levels "female","male": 2 1 1 2
#> $ age : num 33 18 24 26
#> $ city : chr "Taipei" "Taipei" "Hsinchu" "Taichung"
All data structures above are objects. They apply different methods and saved as different type internally.
| object | type | class |
|---|---|---|
| c(1, 2.5, 3) | double | numeric |
| c(“male”, “female”, “female”, “male”) | character | character |
| factor(c(“male”, “female”, “female”, “male”)) | integer | factor |
| matrix(1:9, nrow = 3) | integer | matrix |
| list(a = 1:10, b = c(“apple”, “banana”)) | list | list |
| data.frame(a = 1, b = “z”) | list | data.frame |
To understand computations in R, two slogans are helpful:
- Everything that exists is an object.
- Everything that happens is a function call.
John Chambers
`+`
#> function (e1, e2) .Primitive("+")
`<-`
#> .Primitive("<-")
`[`
#> .Primitive("[")
`c`
#> function (..., recursive = FALSE) .Primitive("c")A typical function in R may look like:
f <- function(par1, par2, ...) {
# Some magic happened
return(sth) # return something
}return to specify the return value, the return value will be the last expression inside the function.The basc structure of conditional execution in R is:
if (an expression returns TRUE or FALSE) {
# do something
} else if (another expression returns TRUE or FALSE) {
# do something
} else {
# do something
}Iterate items in R.
# iterate a character vector
for (i in c("a", "b")) {
print(i)
}#> [1] "a"
#> [1] "b"
# nested loop
m <- matrix(numeric(), nrow = 2, ncol = 2)
for (i in 1:nrow(m)) {
for (j in 1:ncol(m)) {
m[i, j] <- i * j
}
}
m#> [,1] [,2]
#> [1,] 1 2
#> [2,] 2 4
tryCatch({
result <- expr
# If you want to use more than one
# R expression in the "try" part then you'll have to
# use curly brackets.
# 'tryCatch()' will return the last evaluated expression
# in case the "try" part was completed successfully
}, warning = function(w) {
message("Here's the original warning message:")
message(w)
# Choose a return value in case of warning
return(NULL)
}, error = function(e) {
message("Here's the original error message:")
message(e)
# Choose a return value in case of error
return(NA)
}, finally {
message("Some other message at the end")
# finally:
# Here goes everything that should be executed at the end,
# regardless of success or error.
# Usually it's used for releasing resources.
# If you want more than one expression to be executed, then you
# need to wrap them in curly brackets ({...}); otherwise you could
# just have written 'finally=<expression>'
})https://stackoverflow.com/questions/12193779/how-to-write-trycatch-in-r/12195574#12195574
Pipe argument to right-hand side with %>%
x %>% f is equivalent to f(x)x %>% f(y) is equivalent to f(x, y)x %>% f %>% g %>% h is equivalent to h(g(f(x)))x %>% f(y, .) is equivalent to f(y, x)x %>% f(y, z = .) is equivalent to f(y, z = x)library(magrittr)
iris %>% head()"Ceci n'est pas une pipe" %>% gsub("une", "un", .)A valid HTTP request includes four things:
{httr}{RCurl}download.file()make.socket / read.socket / close.socketGET(), POST(), HEAD()…http_status: Translate http status code HTTP status codestatus_codeset_cookies(): set cooklesadd_headers(): add additional headers to a request.headers(): Access response headerscontent(): Retrieve the contents of a request
Use GET() to request data from a specific resource
## Not Run
library(httr)
res <- GET(
url = "http://httpbin.org/get",
add_headers(a = 1, b = 2),
set_cookies(c = 1, d = 2),
query = list(q="hihi")
)
content(res, as = "text", encoding = "UTF-8")
content(res, as = "parsed", encoding = "UTF-8")library(magrittr)
library(httr)
library(rvest)
#> Loading required package: xml2
## Connection
url <- "https://www.ptt.cc/bbs/Gossiping/index.html"
res <- GET(url,
set_cookies(over18="1")) # over18 cookie
## Parsing
doc <- res %>%
content(as = "text", encoding = "UTF-8") %>%
read_html
doc %>%
html_nodes(css = ".title a") %>%
html_text()
#> [1] "[問卦] 頭文字D會拍第二集嗎?"
#> [2] "[問卦] 看新唐人電視的都是哪些人?"
#> [3] "[新聞] 學當在野黨!藍委「同心營」 選6星渡假村"
#> [4] "[FB] 林昶佐接受瑞士SWR電台專訪"
#> [5] "[問卦] 寵物帶來樂趣與辛苦度的圖"
#> [6] "[問卦] 母性動物護幼子的行為真的很偉大"
#> [7] "[問卦] 有沒有一代歌姬白冰冰的八卦"
#> [8] "[問卦] 有沒有10次車禍999999999次快的八卦"
#> [9] "[問卦] 若沒有8+9的活動可以參加,台灣社會變成?"
#> [10] "[問卦] 全家能買什麼?"
#> [11] "[問卦] 有沒有同事愛裝怪聲的八卦?"
#> [12] "[問卦] 日本人很不耐熱嗎"
#> [13] "[新聞] 孔傑榮鼓勵馬英九 訪問中國大陸及美國"
#> [14] "[問卦] 校園下一個該解禁的項目"
#> [15] "[公告] 八卦板板規(2016.02.16)"
#> [16] "[協尋]行車記錄器-5/14承德路5段與基河路交叉口"
#> [17] "Fw: [申請] 發文門檻提高 5/19-5/26"
#> [18] "[公告] 五月份置底閒聊文"
#> [19] "[公告] 數字類跟風問卦視為鬧板"set_cookies to retreve a request.library(httr)
url <- "https://www.ptt.cc/bbs/Gossiping/index.html"
res <- GET(url,
set_cookies('over18'='1')) # over18 cookieurl <- "https://www.ptt.cc/bbs/Gossiping/index.html"
res <- GET(url,
set_cookies('over18'='1')) # over18 cookie
# Get an informative description:
http_status(res)
#> $category
#> [1] "Success"
#>
#> $reason
#> [1] "OK"
#>
#> $message
#> [1] "Success: (200) OK"
# Or just access the raw code:
res$status_code
#> [1] 200
status_code(res)
#> [1] 200
# highly recommend using one of these functions whenever you're using httr inside a function to make sure you find out about errors as soon as possible.
warn_for_status(res)
stop_for_status(res)## Not Run
library(httr)
library(rvest)
res <- POST(url = "http://httpbin.org/post",
add_headers(a = 1, b = 2),
set_cookies(c = 1, d = 2),
body = list(x = "hello",
y = "hihi"))
content(res, as = "text", encoding = "UTF-8")
content(res, as = "parsed", encoding = "UTF-8")Try to post a message in App Engine GuestBook
Sometime you may need to provide appropriate HTTP header fields with add_headers() to make a request.
3 ways to access the body of the request:
httr::content()res <- GET("https://www.ptt.cc/bbs/NBA/M.1463302851.A.8D7.html")(bin <- content(res, as = "raw")) %>% head(200) # raw (binary) vector#> [1] 3c 21 44 4f 43 54 59 50 45 20 68 74 6d 6c 3e 0a 3c 68 74 6d 6c 3e 0a
#> [24] 09 3c 68 65 61 64 3e 0a 09 09 3c 6d 65 74 61 20 63 68 61 72 73 65 74
#> [47] 3d 22 75 74 66 2d 38 22 3e 0a 09 09 0a 0a 3c 6d 65 74 61 20 6e 61 6d
#> [70] 65 3d 22 76 69 65 77 70 6f 72 74 22 20 63 6f 6e 74 65 6e 74 3d 22 77
#> [93] 69 64 74 68 3d 64 65 76 69 63 65 2d 77 69 64 74 68 2c 20 69 6e 69 74
#> [116] 69 61 6c 2d 73 63 61 6c 65 3d 31 22 3e 0a 0a 3c 74 69 74 6c 65 3e 5b
#> [139] e8 a8 8e e8 ab 96 5d 20 e6 9e 97 e6 9b b8 e8 b1 aa e5 a6 82 e6 9e 9c
#> [162] e5 8a a0 e5 85 a5 e9 a6 ac e5 88 ba e8 83 bd e5 be 97 e5 88 b0 e5 a4
#> [185] 9a e5 b0 91 e4 b8 8a e5 a0 b4 e6 99 82 e9 96 93
# writeBin(bin, "myfile.txt") # the highest fidelity way of saving files to diskcontent(res, as = "text", encoding = "UTF-8") %>% # accesses the body as a character vector
`Encoding<-`("UTF-8")#> [1] "<!DOCTYPE html>\n<html>\n\t<head>\n\t\t<meta charset=\"utf-8\">\n\t\t\n\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n\n<title>[討論] 林書豪如果加入馬刺能得到多少上場時間? - 看板 NBA - 批踢踢實業坊</title>\n<meta name=\"robots\" content=\"all\">\n<meta name=\"keywords\" content=\"Ptt BBS 批踢踢\">\n<meta name=\"description\" content=\"\n\n\n最近有傳言說林書豪可能會加入馬刺\n\n\">\n<meta property=\"og:site_name\" content=\"Ptt 批踢踢實業坊\">\n<meta property=\"og:title\" content=\"[討論] 林書豪如果加入馬刺能得到多少上場時間?\">\n<meta property=\"og:description\" content=\"\n\n\n最近有傳言說林書豪可能會加入馬刺\n\n\">\n<link rel=\"canonical\" href=\"https://www.ptt.cc/bbs/NBA/M.1463302851.A.8D7.html\">\n\n<link rel=\"stylesheet\" type=\"text/css\" href=\"//images.ptt.cc/v2.17/bbs-common.css\">\n<link rel=\"stylesheet\" type=\"text/css\" href=\"//images.ptt.cc/v2.17/bbs-base.css\" media=\"screen\">\n<link rel=\"stylesheet\" type=\"text/css\" href=\"//images.ptt.cc/v2.17/bbs-custom.css\">\n<link rel=\"stylesheet\" type=\"text/css\" href=\"//images.ptt.cc/v2.17/pushstream.css\" media=\"screen\">\n<link rel=\"stylesheet\" type=\"text/css\" href=\"//images.ptt.cc/v2.17/bbs-print.css\" media=\"print\">\n\n\n<script src=\"//ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js\"></script>\n<script src=\"//images.ptt.cc/v2.17/bbs.js\"></script>\n\n\n\t\t\n\n<script type=\"text/javascript\">\n\n var _gaq = _gaq || [];\n _gaq.push(['_setAccount', 'UA-32365737-1']);\n _gaq.push(['_setDomainName', 'ptt.cc']);\n _gaq.push(['_trackPageview']);\n\n (function() {\n var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;\n ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';\n var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);\n })();\n\n</script>\n\n\n\t</head>\n <body>\n\t\t\n<div id=\"fb-root\"></div>\n<script>(function(d, s, id) {\nvar js, fjs = d.getElementsByTagName(s)[0];\nif (d.getElementById(id)) return;\njs = d.createElement(s); js.id = id;\njs.src = \"//connect.facebook.net/en_US/all.js#xfbml=1\";\nfjs.parentNode.insertBefore(js, fjs);\n}(document, 'script', 'facebook-jssdk'));</script>\n\n<div id=\"topbar-container\">\n\t<div id=\"topbar\" class=\"bbs-content\">\n\t\t<a id=\"logo\" href=\"/\">批踢踢實業坊</a>\n\t\t<span>›</span>\n\t\t<a class=\"board\" href=\"/bbs/NBA/index.html\"><span class=\"board-label\">看板 </span>NBA</a>\n\t\t<a class=\"right small\" href=\"/about.html\">關於我們</a>\n\t\t<a class=\"right small\" href=\"/contact.html\">聯絡資訊</a>\n\t</div>\n</div>\n<div id=\"navigation-container\">\n\t<div id=\"navigation\" class=\"bbs-content\">\n\t\t<a class=\"board\" href=\"/bbs/NBA/index.html\">返回看板</a>\n\t\t<div class=\"bar\"></div>\n\t\t<div class=\"share\">\n\t\t\t<span>分享</span>\n\t\t\t<div class=\"fb-like\" data-send=\"false\" data-layout=\"button_count\" data-width=\"90\" data-show-faces=\"false\" data-href=\"http://www.ptt.cc/bbs/NBA/M.1463302851.A.8D7.html\"></div>\n\n\t\t\t<div class=\"g-plusone\" data-size=\"medium\"></div>\n<script type=\"text/javascript\">\nwindow.___gcfg = {lang: 'zh-TW'};\n(function() {\nvar po = document.createElement('script'); po.type = 'text/javascript'; po.async = true;\npo.src = 'https://apis.google.com/js/plusone.js';\nvar s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s);\n})();\n</script>\n\n\t\t</div>\n\t</div>\n</div>\n<div id=\"main-container\">\n <div id=\"main-content\" class=\"bbs-screen bbs-content\"><div class=\"article-metaline\"><span class=\"article-meta-tag\">作者</span><span class=\"article-meta-value\">magneto5566 (萬磁王5566_麥寮法斯賓達)</span></div><div class=\"article-metaline-right\"><span class=\"article-meta-tag\">看板</span><span class=\"article-meta-value\">NBA</span></div><div class=\"article-metaline\"><span class=\"article-meta-tag\">標題</span><span class=\"article-meta-value\">[討論] 林書豪如果加入馬刺能得到多少上場時間?</span></div><div class=\"article-metaline\"><span class=\"article-meta-tag\">時間</span><span class=\"article-meta-value\">Sun May 15 17:00:48 2016</span></div>\n\n\n最近有傳言說林書豪可能會加入馬刺\n\n不管對林書豪或對球迷來說馬刺都是個很棒的選擇\n\n馬刺今年例行賽戰績高達67勝,主力陣容維持的話明年應該依舊有60勝左右\n\n雖然波波維奇用兵如神,但林書豪在馬刺到底能獲得多少上場時間呢\n\n林書豪應該是替補PG和SG的位置\n\n單看馬刺和雷霆的系列賽\n\n馬刺的後場部分主要由4位球員分擔時間\n\n先發: Danny Green , Tony Parker\n\n替補: Manu Ginobili, Patty Mills\n\n\n理論上 林書豪最有機會的是Ginobili的位置\n\n畢竟鬼切已經38歲,下季會不會回來都還不確定\n\n林書豪的切入能力雖然沒有巔峰的鬼切犀利\n\n但是大勝38歲鬼切應該是沒問題\n\n相信波波維奇很了解這一點\n\n不知道大家怎麼看呢?\n\n\n--\n<span class=\"f2\">※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 140.112.71.9\n</span><span class=\"f2\">※ 文章網址: <a href=\"https://www.ptt.cc/bbs/NBA/M.1463302851.A.8D7.html\" target=\"_blank\" rel=\"nofollow\">https://www.ptt.cc/bbs/NBA/M.1463302851.A.8D7.html</a>\n</span><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">sss1234 </span><span class=\"f3 push-content\">: 87分鐘</span><span class=\"push-ipdatetime\"> 05/15 17:02\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">h22212247888</span><span class=\"f3 push-content\">: 8.7分鐘</span><span class=\"push-ipdatetime\"> 05/15 17:02\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">rick </span><span class=\"f3 push-content\">: 等七月再來分析好嗎?</span><span class=\"push-ipdatetime\"> 05/15 17:02\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">demonh311 </span><span class=\"f3 push-content\">: 當然是當基石</span><span class=\"push-ipdatetime\"> 05/15 17:02\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">arcss </span><span class=\"f3 push-content\">: 前半年冰凍,後半年15分鐘/場</span><span class=\"push-ipdatetime\"> 05/15 17:03\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">hunt5566 </span><span class=\"f3 push-content\">: 呵呵 波波又藏招不讓豪鬼上囉</span><span class=\"push-ipdatetime\"> 05/15 17:04\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">Xenogamer </span><span class=\"f3 push-content\">: 0</span><span class=\"push-ipdatetime\"> 05/15 17:04\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">love1500274 </span><span class=\"f3 push-content\">: Pop會讓書豪取代跑車先發 書豪會拿下年度mvp</span><span class=\"push-ipdatetime\"> 05/15 17:05\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">littlegreen </span><span class=\"f3 push-content\">: 馬刺需要切入沒錯 但是切傳能力 這書豪比較弱</span><span class=\"push-ipdatetime\"> 05/15 17:06\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">噓 </span><span class=\"f3 hl push-userid\">r30385 </span><span class=\"f3 push-content\">: 8.7秒</span><span class=\"push-ipdatetime\"> 05/15 17:08\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">donnylee </span><span class=\"f3 push-content\">: Mills也差不多要退了,40歲</span><span class=\"push-ipdatetime\"> 05/15 17:12\n</span></div> Mills 27歲而已\n<span class=\"f2\">※ 編輯: magneto5566 (140.112.71.9), 05/15/2016 17:14:25\n</span><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">darkfeather </span><span class=\"f3 push-content\">: 美媒建議變傳言,這進化速度還真快</span><span class=\"push-ipdatetime\"> 05/15 17:15\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">donnylee </span><span class=\"f3 push-content\">: 喔,我記錯了,那是哪一個40?</span><span class=\"push-ipdatetime\"> 05/15 17:18\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">Mooooose </span><span class=\"f3 push-content\">: Popo黑掉的所花的時間會比較多嗎?</span><span class=\"push-ipdatetime\"> 05/15 17:18\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">erotica </span><span class=\"f3 push-content\">: 阿豪打1 2號都可以 還可以各種大小鎖防守</span><span class=\"push-ipdatetime\"> 05/15 17:21\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">噓 </span><span class=\"f3 hl push-userid\">king181239 </span><span class=\"f3 push-content\">: 去專版問啊</span><span class=\"push-ipdatetime\"> 05/15 17:21\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">king181239 </span><span class=\"f3 push-content\">: 他們會說超過35分鐘</span><span class=\"push-ipdatetime\"> 05/15 17:21\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">erotica </span><span class=\"f3 push-content\">: 屌打上述4個人</span><span class=\"push-ipdatetime\"> 05/15 17:21\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">king181239 </span><span class=\"f3 push-content\">: 事實上只能出賽20幾</span><span class=\"push-ipdatetime\"> 05/15 17:22\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">噓 </span><span class=\"f3 hl push-userid\">s925407 </span><span class=\"f3 push-content\">: 怎麼會有人說Mills40歲啦,一日球迷也沒這麼不專業</span><span class=\"push-ipdatetime\"> 05/15 17:22\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">s925407 </span><span class=\"f3 push-content\">: 啦</span><span class=\"push-ipdatetime\"> 05/15 17:22\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">a10141013 </span><span class=\"f3 push-content\">: 馬刺大約20-25分鐘 然後popo被說冰箱</span><span class=\"push-ipdatetime\"> 05/15 17:24\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">dream1285201</span><span class=\"f3 push-content\">: 一個球員會黑掉大多跟他的球迷有關</span><span class=\"push-ipdatetime\"> 05/15 17:24\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">smtp </span><span class=\"f3 push-content\">: 重點是馬刺想出多少爭取豪哥?</span><span class=\"push-ipdatetime\"> 05/15 17:28\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">kl015013 </span><span class=\"f3 push-content\">: 20</span><span class=\"push-ipdatetime\"> 05/15 17:29\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">bycarbird </span><span class=\"f3 push-content\">: 0,因為書豪不是波波的菜</span><span class=\"push-ipdatetime\"> 05/15 17:30\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">MoMovincent </span><span class=\"f3 push-content\">: 那個說40歲的應該是要說a米吧</span><span class=\"push-ipdatetime\"> 05/15 17:30\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">s95115260 </span><span class=\"f3 push-content\">: 馬刺怎麼可能要球權的球員</span><span class=\"push-ipdatetime\"> 05/15 17:31\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">love1500274 </span><span class=\"f3 push-content\">: 他想說的是Miller 打成Mills lol</span><span class=\"push-ipdatetime\"> 05/15 17:32\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">kiwi1220 </span><span class=\"f3 push-content\">: 40是 ,Miller</span><span class=\"push-ipdatetime\"> 05/15 17:33\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">miler22020 </span><span class=\"f3 push-content\">: 20分鐘 球權不多 然後popo從此黑掉</span><span class=\"push-ipdatetime\"> 05/15 17:36\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">scatology </span><span class=\"f3 push-content\">: 5 垃圾時間</span><span class=\"push-ipdatetime\"> 05/15 17:36\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">smtp </span><span class=\"f3 push-content\">: 10M簽的話, 上場時間不可能只有20分鐘...</span><span class=\"push-ipdatetime\"> 05/15 17:37\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">forker561 </span><span class=\"f3 push-content\">: 第一年不會有上場時間 最快第二年才有上場時間</span><span class=\"push-ipdatetime\"> 05/15 17:38\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">candy9999 </span><span class=\"f3 push-content\">: 去哪都比在黃蜂好</span><span class=\"push-ipdatetime\"> 05/15 17:40\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">DiAbLoE </span><span class=\"f3 push-content\">: popo會變黑人</span><span class=\"push-ipdatetime\"> 05/15 17:41\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">smtp </span><span class=\"f3 push-content\">: 豪哥今年曾說,年薪影響別人對你評價,可見企圖心很強</span><span class=\"push-ipdatetime\"> 05/15 17:41\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">happyalex </span><span class=\"f3 push-content\">: 公道價8分鐘</span><span class=\"push-ipdatetime\"> 05/15 17:43\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">goury </span><span class=\"f3 push-content\">: 千萬不要去馬刺...Linsanity的重點是明星度,而不是</span><span class=\"push-ipdatetime\"> 05/15 17:47\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">goury </span><span class=\"f3 push-content\">: 馬刺這樣只在乎勝利,讓買票進場的人看不到球星的隊</span><span class=\"push-ipdatetime\"> 05/15 17:48\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">loverxa </span><span class=\"f3 push-content\">: 到時不用豪神 就沒人在跟你用兵如神了啦</span><span class=\"push-ipdatetime\"> 05/15 17:49\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">crazywill </span><span class=\"f3 push-content\">: 馬刺不會要阿豪 不用討論了</span><span class=\"push-ipdatetime\"> 05/15 17:50\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">birdman5656 </span><span class=\"f3 push-content\">: 先等波波帶完奧運吧</span><span class=\"push-ipdatetime\"> 05/15 17:51\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">yyyyy </span><span class=\"f3 push-content\">: 馬刺應該以我豪為核心進行重建才對</span><span class=\"push-ipdatetime\"> 05/15 17:53\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">leocheng </span><span class=\"f3 push-content\">: 馬刺喜歡跳投好的 怎麼可能愛豪鬼</span><span class=\"push-ipdatetime\"> 05/15 18:04\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">june0204 </span><span class=\"f3 push-content\">: 還是2000找康利吧</span><span class=\"push-ipdatetime\"> 05/15 18:12\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">HowWhy99 </span><span class=\"f3 push-content\">: 8.7分 不能再多了</span><span class=\"push-ipdatetime\"> 05/15 18:12\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">crazylin924 </span><span class=\"f3 push-content\">: 35分鐘 (一整季</span><span class=\"push-ipdatetime\"> 05/15 18:19\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">噓 </span><span class=\"f3 hl push-userid\">xman262 </span><span class=\"f3 push-content\">: 487秒</span><span class=\"push-ipdatetime\"> 05/15 18:24\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">噓 </span><span class=\"f3 hl push-userid\">ppo7741 </span><span class=\"f3 push-content\">: 8.7分,跟這篇文的分數一樣,不能再高了</span><span class=\"push-ipdatetime\"> 05/15 18:25\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">carey1119 </span><span class=\"f3 push-content\">: 8.7分鐘</span><span class=\"push-ipdatetime\"> 05/15 18:33\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">DurantKevin </span><span class=\"f3 push-content\">: 。。。</span><span class=\"push-ipdatetime\"> 05/15 18:34\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">qq1029qq </span><span class=\"f3 push-content\">: 87分鐘無誤</span><span class=\"push-ipdatetime\"> 05/15 18:40\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">drew9992 </span><span class=\"f3 push-content\">: <a href=\"http://i.imgur.com/0XBGyQU.jpg\" target=\"_blank\" rel=\"nofollow\">http://i.imgur.com/0XBGyQU.jpg</a></span><span class=\"push-ipdatetime\"> 05/15 19:01\n</span></div><div class=\"richcontent\"><img src=\"//i.imgur.com/0XBGyQU.jpg\" alt=\"\" /></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">dai26 </span><span class=\"f3 push-content\">: 若能完全取代馬妞,大概30分鐘,而且可打滿第四節</span><span class=\"push-ipdatetime\"> 05/15 19:20\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">vicyong </span><span class=\"f3 push-content\">: 不要在幻想了</span><span class=\"push-ipdatetime\"> 05/15 19:23\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">gotohikaru </span><span class=\"f3 push-content\">: 第一年不用想</span><span class=\"push-ipdatetime\"> 05/15 19:32\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">waw20002002 </span><span class=\"f3 push-content\">: 8.7分鐘!不能在多了</span><span class=\"push-ipdatetime\"> 05/15 20:13\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">噓 </span><span class=\"f3 hl push-userid\">an565an565 </span><span class=\"f3 push-content\">: 後衛來馬刺都要被冰一整季</span><span class=\"push-ipdatetime\"> 05/15 21:07\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">trylovetom </span><span class=\"f3 push-content\">: Parker不是被豪哥哥電?</span><span class=\"push-ipdatetime\"> 05/15 21:09\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">smtp </span><span class=\"f3 push-content\">: Parker是輸在年紀, 巔峰時不可能會被豪哥電~</span><span class=\"push-ipdatetime\"> 05/15 21:12\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">噓 </span><span class=\"f3 hl push-userid\">Yginger1 </span><span class=\"f3 push-content\">: 42分鐘 全場打好打滿</span><span class=\"push-ipdatetime\"> 05/15 21:19\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">pickoff </span><span class=\"f3 push-content\">: 馬刺不收。問下一隊謝謝</span><span class=\"push-ipdatetime\"> 05/15 21:22\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">mmshlovebob </span><span class=\"f3 push-content\">: 8.7分鐘 不能再多了</span><span class=\"push-ipdatetime\"> 05/15 21:36\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">roc074 </span><span class=\"f3 push-content\">: 8.7時</span><span class=\"push-ipdatetime\"> 05/15 22:04\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">噓 </span><span class=\"f3 hl push-userid\">deathgale </span><span class=\"f3 push-content\">: 等加盟了再來討論,記者都自以為球團老闆...</span><span class=\"push-ipdatetime\"> 05/15 22:38\n</span></div><div class=\"push\"><span class=\"hl push-tag\">推 </span><span class=\"f3 hl push-userid\">squall410339</span><span class=\"f3 push-content\">: 快去啦馬刺黑掉就是爽</span><span class=\"push-ipdatetime\"> 05/15 23:29\n</span></div><div class=\"push\"><span class=\"f1 hl push-tag\">→ </span><span class=\"f3 hl push-userid\">raku </span><span class=\"f3 push-content\">: 一開始一場應該是不到10分鐘...</span><span class=\"push-ipdatetime\"> 05/16 00:34\n</span></div></div>\n \n <div id=\"article-polling\" data-pollurl=\"/poll/NBA/M.1463302851.A.8D7.html?cacheKey=2052-14787853&offset=7844&offset-sig=05b0d881a2af63a68de9ce2d0f1cda56eec2d548\" data-longpollurl=\"/v1/longpoll?id=5ef93f8aa82d4cf930eb27310d147571cf13936a\" data-offset=\"7844\"></div>\n \n\n \n</div>\n\n </body>\n</html>\n"
?httr::contentcontent(res, as = "parsed")#> {xml_document}
#> <html>
#> [1] <head>\n\t\t<meta charset="utf-8"/>\n\t\t\n\n<meta name="viewport" c ...
#> [2] <body>\n\t\t\n<div id="fb-root"/>\n<script><![CDATA[(function(d, s, ...
Encoding(), iconv() and Sys.setlocale() in Windows (non-UTF8 system).iconvlist() to show all encodings availabe.?Encoding ?iconv
## check out your system locale
Sys.getlocale()
#> [1] "en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8"
aa <- "你好嗎"
Encoding(aa)
#> [1] "UTF-8"
charToRaw(aa)
#> [1] e4 bd a0 e5 a5 bd e5 97 8e
(aa_big5 <- iconv(aa, from = "UTF-8", to = "Big5"))
#> [1] "\xa7A\xa6n\xb6\xdc"
Encoding(aa_big5)
#> [1] "unknown"
harToRaw(aa_big5)
#> [1] a7 41 a6 6e b6 dc# On Windows
Sys.getlocale()
#> "CP950"
aa <- "你好嗎"
Encoding(aa)
#> "unknown"
aa_utf8 <- iconv(aa, from = "Big5", to = "UTF-8")
Encoding(aa_utf8)
#> "UTF-8"URL?par1=val1&par2=val2
You can assign query parameter with query()
res1 <- GET(
"http://ecshweb.pchome.com.tw/search/v3.3/all/results?q=sony&page=1&sort=rnk/dc"
)
res2 <- GET("http://ecshweb.pchome.com.tw/search/v3.3/all/results",
query = list(q="sony", page="1", sort="rnk/dc"))URLencode(" ")#> [1] "%20"
greeting = "你好嗎我很好"
greeting_enc = URLencode(greeting)
greeting_enc#> [1] "%E4%BD%A0%E5%A5%BD%E5%97%8E%E6%88%91%E5%BE%88%E5%A5%BD"
URLdecode(greeting_enc)#> [1] "你好嗎我很好"
paste0("hihi", greeting)#> [1] "hihi你好嗎我很好"
paste0("hihi", greeting, 1:3)#> [1] "hihi你好嗎我很好1" "hihi你好嗎我很好2" "hihi你好嗎我很好3"
paste("hihi", greeting, 1:3, sep = " ", collapse = ",")#> [1] "hihi 你好嗎我很好 1,hihi 你好嗎我很好 2,hihi 你好嗎我很好 3"
sprintf("%s,%s嗎?", "hihi", greeting)#> [1] "hihi,你好嗎我很好嗎?"
rvest::html_node)jsonlite::fromJSON)rvest::html_table, XML::readHtmlTable)XML::xmlToDataFrame)A web scraper designed to work with magrittr.
read_html()html_nodes(doc, css = "<css selector>")html_nodes(doc, xpath = "<css selector>")html_name(): the name of the taghtml_text(): all text inside the taghtml_attr(): contents of a single attributehtml_attrs(): all attributeshtml_table: parse tables into data framesGET(url) %>% content(as="text") %>% rvest::read_html()<a href = "www.meetup.com/Taiwan-R">
Taiwan R User Group Website
</a>ahref with value "www.meetup.com/Taiwan-R"Taiwan R User Group Websitelibrary(magrittr)
doc = readLines("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html") %>%
paste(collapse = "\n")
cat(doc)#> <!DOCTYPE HTML>
#> <html lang="zh-TW">
#> <head>
#> <meta charset="UTF-8">
#> <title>Document</title>
#> </head>
#> <html>
#> <body>
#> <div id='title' class='character'>
#> <a class="link" href="http://data-sci.info/r-crawler-101/">data-sci.info</a>
#> </div>
#> <div id='summary' class='character'>
#> <span class='title'>Number:</span>
#> <span class='number'>2</span>
#> </div>
#> <div id='table1' class='info'>
#> <table style="border-collapse: collapse; border: 1px solid black;">
#> <tr>
#> <th>Name</th>
#> <th>Gender</th>
#> <th>Age</th>
#> </tr>
#> <tr>
#> <td>Alice</td>
#> <td>Female</td>
#> <td>24</td>
#> </tr>
#> <tr>
#> <td>Jane</td>
#> <td>Female</td>
#> <td>26</td>
#> </tr>
#> </table>
#> </div>
#>
#> <h3>傷心排行榜</h3>
#> <div id='table2' class='info'>
#> <table style="border-collapse: collapse; border: 1px solid black;">
#> <tr>
#> <th>姓名</th>
#> <th>年齡</th>
#> </tr>
#> <tr>
#> <td>白素貞</td>
#> <td>15</td>
#> </tr>
#> <tr>
#> <td>聶小倩</td>
#> <td>32</td>
#> </tr>
#> <tr>
#> <td>祝英台</td>
#> <td>54</td>
#> </tr>
#> </table>
#> </div>
#> </body>
#> </html>
library(rvest)
doc <- read_html("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html")
doc#> {xml_document}
#> <html>
#> [1] <head>\n <meta charset="UTF-8"/>\n <title>Document</title>\n</ ...
#> [2] <body>\n <div id="title" class="character">\n <a c ...
class(doc)#> [1] "xml_document" "xml_node"
CSS practice
http://flukeout.github.io/
doc <- read_html("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html")
doc %>%
html_nodes(css = ".character") %>%
html_text#> [1] "\n data-sci.info\n "
#> [2] "\n Number:\n 2\n "
doc %>%
html_nodes(css = "#title > .link") %>%
html_text#> [1] "data-sci.info"
doc <- read_html("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html")
doc %>%
html_nodes(xpath = "//*[@class='character']") %>%
html_text#> [1] "\n data-sci.info\n "
#> [2] "\n Number:\n 2\n "
doc %>%
html_nodes(xpath = "//div[@id='title']/a") %>%
html_text#> [1] "data-sci.info"
node = doc %>%
html_nodes(css = "#summary") %>%
html_name
node#> [1] "div"
link = doc %>%
html_nodes(xpath = "/html/body/div[@id='title']/a") %>%
html_attr("href")
link#> [1] "http://data-sci.info/r-crawler-101/"
use package xmlview to parse XML content and extract elements.
rvest::html_table()(html_table() is temperarily not working with non-ascii data on Windows.)
students <- read_html("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html") %>%
html_nodes(css = "#table1>table") %>%
html_table()
students#> [[1]]
#> Name Gender Age
#> 1 Alice Female 24
#> 2 Jane Female 26
XML::readHTMLTable()res_text <- GET("http://leoluyi.github.io/RCrawler101_201605_Week2/resources/data/demo.html") %>%
content("text", encoding = "UTF-8") %>%
`Encoding<-`("UTF-8")
# read and extract tables from html string
tables <- res_text %>% XML::readHTMLTable(encoding = "UTF-8")str(tables) # take a look
View(tables[[2]]) # take a look | 姓名 | 年齡 |
|---|---|
| 白素貞 | 15 |
| 聶小倩 | 32 |
| 祝英台 | 54 |
Parse XML table with XML::xmlToDataFrame()
Yun can parse JSON with jsonlite in R
jsonlitelibrary(jsonlite)
library(magrittr)
res <- GET("http://ecshweb.pchome.com.tw/search/v3.3/all/results?q=sony&page=1&sort=rnk/dc")jsonlite::fromJSON()res_df <- content(res, as = "text", encoding = "UTF-8") %>%
fromJSON() %>%
.$prods # equivelent to (function(x) {x$prods})head(res_df)
#> Id cateId
#> 1 DYAD2P-A9006JOZX DYAD2P
#> 2 DYAD2R-A9006XKSI DYAD2R
#> 3 DPAE03-A9006RR9X DPAE0V
#> 4 DYAD2O-A9006JDQE DYAD2O
#> 5 DYAD2O-A9006J87V DYAD2O
#> 6 DGBT30-A900649ND DYAQA6
#> picS
#> 1 /pic/v1/data/item/201510/D/Y/A/D/2/P/sDYAD2P-A9006JOZX000_5614b4cc793f6.jpg
#> 2 /pic/v1/data/item/201604/D/Y/A/D/2/R/sDYAD2R-A9006XKSI000_56fe3f0b20c2e.jpg
#> 3 /pic/v1/data/item/201601/D/P/A/E/0/3/sDPAE03-A9006RR9X000_56970708b9924.jpg
#> 4 /pic/v1/data/item/201510/D/Y/A/D/2/O/sDYAD2O-A9006JDQE000_560e037b08db2.jpg
#> 5 /pic/v1/data/item/201509/D/Y/A/D/2/O/sDYAD2O-A9006J87V000_560b9b7cb48ad.jpg
#> 6 /pic/v1/data/item/201505/D/G/B/T/3/0/sDGBT30-A900649ND000_5567e78207366.jpg
#> picB
#> 1 /pic/v1/data/item/201510/D/Y/A/D/2/P/DYAD2P-A9006JOZX000_5614b4cc76ca5.jpg
#> 2 /pic/v1/data/item/201605/D/Y/A/D/2/R/DYAD2R-A9006XKSI000_57426fcf0b981.jpg
#> 3 /pic/v1/data/item/201605/D/P/A/E/0/3/DPAE03-A9006RR9X000_573e6e365d2c4.jpg
#> 4 /pic/v1/data/item/201604/D/Y/A/D/2/O/DYAD2O-A9006JDQE000_57198b2dc111b.jpg
#> 5 /pic/v1/data/item/201605/D/Y/A/D/2/O/DYAD2O-A9006J87V000_57426e333e83e.jpg
#> 6 /pic/v1/data/item/201604/D/G/B/T/3/0/DGBT30-A900649ND000_5715927991a26.jpg
#> name
#> 1 Sony Xperia Z5 Compact E5823 4.6吋八核輕旗艦
#> 2 SONY Xperia Z5 Premium
#> 3 Sony 行動微型投影機 MP-CL1
#> 4 SONY Xperia Z5
#> 5 SONY Xperia Z5
#> 6 SONY SBH70 (防水IP57)無線藍牙耳機
#> describe price
#> 1 送9H玻璃保貼+保護套 Sony Xperia Z5 Compact E5823 4.6吋八核輕旗艦 13880
#> 2 ▼送32G卡+手機立架+玻璃貼SONY Xperia Z5 Premium 19900
#> 3 Sony 行動微型投影機 MP-CL1 11900
#> 4 超值▼送行動電源+保護貼+保護套▼SONY Xperia Z5 5.2吋美型防水旗艦機 16888
#> 5 ★送保貼+支架+觸控筆吊飾★SONY Xperia Z5 16990
#> 6 SONY SBH70 (防水IP57)無線藍牙耳機 2188
#> author brand publishDate isPick
#> 1 0
#> 2 0
#> 3 0
#> 4 0
#> 5 0
#> 6 0httr auto parse json as listres_list <- content(res, as = "parsed")
# str(res_list)
res_list$prods[[1]]#> $Id
#> [1] "DYAD2P-A9006JOZX"
#>
#> $cateId
#> [1] "DYAD2P"
#>
#> $picS
#> [1] "/pic/v1/data/item/201510/D/Y/A/D/2/P/sDYAD2P-A9006JOZX000_5614b4cc793f6.jpg"
#>
#> $picB
#> [1] "/pic/v1/data/item/201510/D/Y/A/D/2/P/DYAD2P-A9006JOZX000_5614b4cc76ca5.jpg"
#>
#> $name
#> [1] "Sony Xperia Z5 Compact E5823 4.6吋八核輕旗艦"
#>
#> $describe
#> [1] "送9H玻璃保貼+保護套 Sony Xperia Z5 Compact E5823 4.6吋八核輕旗艦"
#>
#> $price
#> [1] 13880
#>
#> $author
#> [1] ""
#>
#> $brand
#> [1] ""
#>
#> $publishDate
#> [1] ""
#>
#> $isPick
#> [1] 0
# str(res_list$prods)
res_df2 = data.frame()
for (i in 1:length(res_list$prods)) {
res_df2 = rbind(res_df2,
data.frame(res_list$prods[[i]],
stringsAsFactors = FALSE))
}
identical(res_df, res_df2)#> [1] TRUE
(better use do.call + rbind)
do.call + rbinddata.table::rbindlistdo.call + rbind# make a list of dataframes
df_list <- lapply(res_list$prods, as.data.frame, stringsAsFactors = FALSE)
# combine dataframes
res_df3 = do.call(rbind, df_list)
identical(res_df, res_df3)#> [1] TRUE
# str(res_df3)data.table::rbindlistlibrary(data.table)
res_df4 = rbindlist(df_list)
head(res_df4)#> Id cateId
#> 1: DYAD2P-A9006JOZX DYAD2P
#> 2: DYAD2R-A9006XKSI DYAD2R
#> 3: DPAE03-A9006RR9X DPAE0V
#> 4: DYAD2O-A9006JDQE DYAD2O
#> 5: DYAD2O-A9006J87V DYAD2O
#> 6: DGBT30-A900649ND DYAQA6
#> picS
#> 1: /pic/v1/data/item/201510/D/Y/A/D/2/P/sDYAD2P-A9006JOZX000_5614b4cc793f6.jpg
#> 2: /pic/v1/data/item/201604/D/Y/A/D/2/R/sDYAD2R-A9006XKSI000_56fe3f0b20c2e.jpg
#> 3: /pic/v1/data/item/201601/D/P/A/E/0/3/sDPAE03-A9006RR9X000_56970708b9924.jpg
#> 4: /pic/v1/data/item/201510/D/Y/A/D/2/O/sDYAD2O-A9006JDQE000_560e037b08db2.jpg
#> 5: /pic/v1/data/item/201509/D/Y/A/D/2/O/sDYAD2O-A9006J87V000_560b9b7cb48ad.jpg
#> 6: /pic/v1/data/item/201505/D/G/B/T/3/0/sDGBT30-A900649ND000_5567e78207366.jpg
#> picB
#> 1: /pic/v1/data/item/201510/D/Y/A/D/2/P/DYAD2P-A9006JOZX000_5614b4cc76ca5.jpg
#> 2: /pic/v1/data/item/201605/D/Y/A/D/2/R/DYAD2R-A9006XKSI000_57426fcf0b981.jpg
#> 3: /pic/v1/data/item/201605/D/P/A/E/0/3/DPAE03-A9006RR9X000_573e6e365d2c4.jpg
#> 4: /pic/v1/data/item/201604/D/Y/A/D/2/O/DYAD2O-A9006JDQE000_57198b2dc111b.jpg
#> 5: /pic/v1/data/item/201605/D/Y/A/D/2/O/DYAD2O-A9006J87V000_57426e333e83e.jpg
#> 6: /pic/v1/data/item/201604/D/G/B/T/3/0/DGBT30-A900649ND000_5715927991a26.jpg
#> name
#> 1: Sony Xperia Z5 Compact E5823 4.6吋八核輕旗艦
#> 2: SONY Xperia Z5 Premium
#> 3: Sony 行動微型投影機 MP-CL1
#> 4: SONY Xperia Z5
#> 5: SONY Xperia Z5
#> 6: SONY SBH70 (防水IP57)無線藍牙耳機
#> describe price
#> 1: 送9H玻璃保貼+保護套 Sony Xperia Z5 Compact E5823 4.6吋八核輕旗艦 13880
#> 2: ▼送32G卡+手機立架+玻璃貼SONY Xperia Z5 Premium 19900
#> 3: Sony 行動微型投影機 MP-CL1 11900
#> 4: 超值▼送行動電源+保護貼+保護套▼SONY Xperia Z5 5.2吋美型防水旗艦機 16888
#> 5: ★送保貼+支架+觸控筆吊飾★SONY Xperia Z5 16990
#> 6: SONY SBH70 (防水IP57)無線藍牙耳機 2188
#> author brand publishDate isPick
#> 1: 0
#> 2: 0
#> 3: 0
#> 4: 0
#> 5: 0
#> 6: 0
stringrstr_replace(), str_replace_all()str_extract(), str_extract_all()library(stringr)
fruits <- c("one apple", "two pears", "three bananas")
str_replace(fruits, "[aeiou]", "-")#> [1] "-ne apple" "tw- pears" "thr-e bananas"
str_replace_all(fruits, "[aeiou]", "-")#> [1] "-n- -ppl-" "tw- p--rs" "thr-- b-n-n-s"
shopping_list <- c("apples x4", "bag of flour", "bag of sugar", "milk x2")
str_extract(shopping_list, "[a-z]+")#> [1] "apples" "bag" "bag" "milk"
str_extract_all(shopping_list, "[a-z]+")#> [[1]]
#> [1] "apples" "x"
#>
#> [[2]]
#> [1] "bag" "of" "flour"
#>
#> [[3]]
#> [1] "bag" "of" "sugar"
#>
#> [[4]]
#> [1] "milk" "x"
行政院環境保護署環境資源資料開放平台提供了一系列的RESTful Api供大家取用,請試著把 紫外線即時監測資料的資料取回來並轉成data.frame
write.csv: write data.frame as csv filedownload.file: save html, jpeg, etcwriteBin: write binary object into diskRSQLite: SQLite connector in Rlibrary(jsonlite)
library(httr)
url = "http://ecshweb.pchome.com.tw/search/v3.3/all/results?q=sony&page=1&sort=rnk/dc"
res_df = GET(url) %>%
content(res, as = "text") %>%
fromJSON() %>%
.$prods # equivelent to (function(x) {x$prods})
write.csv(res_df, "resources/data/pchome.csv", row.names = FALSE)To download a file from the Internet. download.file takes advantage of internet utilities such as curl or wget and may fail if you don’t have any of these utilities in your system.
dest_dir = "resources/data/download"
dir.create(dest_dir, showWarnings = FALSE, recursive = TRUE)
# Download whole HTML file
download.file("https://www.r-project.org/",
file.path(dest_dir, "r-project.org.html"))
# Download image
download.file("https://www.r-project.org/Rlogo.png",
file.path(dest_dir, "Rlogo.png"))
list.files(dest_dir)To write binary data to your local disk.
r = GET("http://opendata.epa.gov.tw/webapi/api/rest/datastore/355000000I-000004/?format=json")
# Set as = "raw" to prevent any character encoding
bin = content(r, as = "raw")
writeBin(bin, "resources/data/download/uv.json")